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[57] ABSTRACT 

A data transmission method quickly and reliably transmits 
data (e.g., a file) from a source to one or more recipients over 
a communications link. The method includes transmitting 
the data, which arc in the form of a plurality of frames, from 
the source over the link to one or more of the recipients until 
all of the plurality of frames have been transmitted over the 
link. While the data are being transmitted by the source, 
acknowledgments from one or more of the recipients are 
received by the source. The acknowledgments indicate 
which frames require retransmission. After all of the frames 
- have been transmitted over the link, a retransmission is 
performed by the source for only those frames which the 
acknowledgments indicate as requiring retransmission. 



10 Claims, 4 Drawing Sheets 



START ^ 



TRANSMIT ALL FRAMES 

AND RECEIVE 
ACKNOWLEDGEMENTS 



—10 



12>/^ANY" 
FRAMES 

REQUIRE XNO 
^RETRANSMISSION^ 

7 



END ^ 



lYES 



RETRANSMIT ONLY 
THOSE FRAMES AND L.14 
RECEIVE 
ACKNOWLEDGEMENTS 



ANY\^16 
FRAMES 
YES X REQUIRE 

.RETRANSMISSION 
(< MAX ?) 



NO 



11/24/2003, EAST Version: 1.4.1 



U.S. Patent 



Sep. 3, 1996 Sheet 1 of 4 



5,553,083 



Q START ^ 



TRANSMIT ALL FRAMES 

AND RECEIVE 
ACKNOWLEDGEMENTS 



— 10 



■^2■^/ANY 
FRAMES 
REQUIRE 
RETRANSMISSION ^ ► 

9 



YES 



RETRANSMIT ONLY 
THOSE FRAMES AND 
RECEIVE 
ACKNOWLEDGEMENTS 



14 



ANYX/-16 
FRAMES 
YES/ REQUIRE ^^mq 
RETRANSMISSION 7^ 
(< MAX ?) 



FIG. 1 



11/24/2003, EAST Version: 1.4.1 




TCP/IP MODEL 



OPERATION 



4/' 



32 



TCP 



UDP 



-30 



INTERNET (IP) 



LINK 



PHYSICAL 



FIG. 3 



11/24/2003, EAST Version: 1.4.1 



U.S. Patent 



Sep. 3, 1996 



Sheet 3 of 4 



5,553,083 



CO 

z 

_J ' 

O 

m 



< 

o 
< 



X 

h- 
z 

-J 
UJ 



id 
o 
o 

m 



CO 

o 
o 

_J 
m 



CM 

o 

2 



CO 



< 



< 



< 



< 



□ 



< 



o 
< 



U. 



o 
o 

m 



DC 
LJJ 

> 
cc 

HI 
CO 



11/24/2003, EAST Version: 1.4.1 



U.S. Patent 



Sep. 3, 1996 



Sheet 4 of 4 



5,553,083 




52 



MEMORY 



r 



54 



I/O 

CONTROLLER 



56 



NETWORK 
INTERFACE 



r 



68 



INPUT 
DEVICE(S) 



T 

58 



HARD 
DISK 
DRIVE 

— r 

62 





FIG. 5 



11/24/2003, EAST Version: 1.4.1 



5,553,083 



METHOD FOR QUICKLY AND RELIABLY 
TRANSMimNG FRAMES OF DATA OVER 
COMMUNICATIONS LINKS 

FIELD OF THE INVENTION 

This invention relates to data transmission, and more 
paiticulariy, to fast and reliable transmission of files from a 
server to one or more clients over communications links. 

BACKGROUND OF THE ESTVENHON 
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Computer networks, such as wide area networks (WANs), 
can provide unicast, multicast, and broadcast services to 
allow conununication between network participants such as 
a server node and one or more client nodes. Broadcast frame 
relay is a service used to communicate over a computer 
network. Multicast IP technology is another service used to 
communicate over a computer network. The term '*broad- 
cast" refers to a server node sending information to all of the 
client nodes connected to the network. The term "multicast" 
refers to a server node sending information to a subset of all 
of the client nodes connected to the network. Broadcast and 
multicast are network capabilities which are relatively new 
over WANs. 

Some information providers desire to deliver information 
electronically by broadcasting or multicasing the informa- 
tion from a server node at a cenu^ location to one or more 
client nodes at remote customer locations via a computer 
network to which the server and the clients are coupled. 
Because broadcast and multicast network services do not 
provide for acknowledgment of the delivered information at 
all, these services can be unreliable. Such unreliability 
generally is undesirable and unacceptable to information 
providers. 

A common protocol suite in use in computer networks is 
TCP/IP, which is the protocol used in the Internet TCP 
stands for Transmission Control Protocol, and IP stands for 
Internet Protocol. TWo file transfer protocols are available in 
association with TCP/IP: (i) File Transfer Protocol (FTP) 
which runs as an application on top of TCP and (ii) TOvial 
FQe TVansfer Protocol (TFTP) which runs on top of UDP. 
UDP stands for User Datagram Protocol. Both TCP and 
UDP are transport protocols which arc responsible for 
end-to-end delivery of information across an internetwork, 
i.e., a network of networks. 

Both FTP and TFTP support point-to-point (i.e., unicast) 
file transfers only. FTP depends on TCP for reliable delivery, 
as TCP is a connection-oriented acknowledged transport 
protocol. TFTP provides its own acknowledgments for reli- 
ability, as it runs on lop of UDP which is a connectionless 
transport service that does not support acknowledgment 

Connection-oriented protocols such as TCP require setup 
and tear-down of virtual circuit coimections. This requires 
significant handshaking to set up the connection, and is not 35 
desired in some networks that are inherently connectionless 
oriented. An example of a network which is inherently 
connectionless oriented is a wireless data networking tech- 
nology called Cellular Digital Packet Data (CDPD). CDPD 
is being deployed rapidly by the cellular carriers in North 60 
America, Latin America, and parts of the Far East CDPD 
utilizes TCP/IP as the primary protocol suite used in the 
network. One feature of the network is channel hopping, 
where data channels attempt to hop away from cellular voice 
channels. Additionally, subscribers to wireless services are 65 
mobile, meaning a particular session may have the trans- 
mission path change as the user enters a new cell area, for 
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example. Both situations defeat the concept of a virtual 
circuit, which attempts to keep a fixed path for the virtual 
circuit after call setup. Additionally, wireless channels are 
usually bandwidth constrained with higher error rates than 
wireline channels, so overhead should be kept to a mini- 
mum. This means that CDPD wireless networks recommend 
applications operate over UDP (the connectionless transport 
layer) only. Thus, TFTP is the file transfer protocol of choice 
for CDPD. 

TFTP breaks files up into packets having 512 bytes of data 
each, and it then sends each data packet one at a time. After 
each data packet is sent, TFTP causes the sending node to 
wait for an acknowledgment from tiie receiving node(s) 
before the sending node is allowed to send the next data 
packet. TFTP is described, for example, in a book by 
Douglas E. Comer (Internetworking with TCP/IP, Volume I 
Principles, Protocols, and Architecture, Second Edition, 
Prentice Hall. 1991. Chapter 23. pages 377-390). 

While acknowledgment is a part of TFTP, the acknowl- 
edgment scheme used in iKlV becomes very inefficient as 
network delay becomes significant and/oris different for two 
or more of the receiving nodes. Like TFTP, some other 
known data transfer mechanisms require packet-by-packet 
acknowledgment, and thus these other mechanisms also are 
relatively slow at transferring the entire amount of data. 

SUMMARY OF THE INVENTION 

It is an object of the present invention to provide both fast 
and reliable transmission of files from a server to one or 
more clients over a communications link. The file transfer 
preferably is a broadcast or a multicast transmission to all or 
a plurality of the clients. In general, file transfer according 
to the invention will not suffer any reduction in speed, 
rehability, or efficiency in the face of link delay, even if that 
delay is significant and/or different for two or more of the 
receiving clients. The invention provides an ideal mecha- 
nism for distributing computer software files electronically. 

The communications link, which couples the server to the 
clients and allows communication therebetween, can be a 
computer network (e.g., a LAN, a WAN, tiie Internet), a 
wireless network (e.g., a packet cellular data network such 
as CDPD), some combination of these types of communi- 
cation mediums, or some other communication medium. 

In accordance with die invention, the clients send 
acknowledgments back to the server as the server is sending 
the data files. The communication is continuous. That is, the 
server does not stop sending the data to wait for acknowl- 
edgments from the clients, but instead the server receives the 
clients' acknowledgments as the server is transmitting the 
data. The clients* acknowledgments indicate to the server 
which particular packets need to be resent. A packet may 
need to be resent because, for example, it was either not 
received or received in error by one or more of the clients. 
After the server has sent the entire amount of data (e.g., the 
entire file) over the link to the clients, the server performs a 
second round of transmissions in which it only rescnds the 
particular packets indicated by the clients as requiring 
retransmission. During this second round, clients only 
acknowledge packets not received, or not received correcUy, 
during the previous round. The process can then continue 
with as many additional rounds of retransmissions as is 
required so that each of the clients correctly receives all of 
the packets. Alternatively, the retransmission rounds can be 
repeated a predetermined number of times, which number 
can be modified (i.e., the number is configurable). Each 
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subsequent round typically involves the transmission of 
fewer packets than the previous round, as only previous 
packets in error are resent. 

This scheme quickly and reliably transfers data from a 
server to one or more clients. It is quick because the server 5 
is allowed to transfer the entire file without stopping at 
packet boundaries to wait for acknowledgments from the 
clients for the packet just sent. That is, data transfer is not 
direcdy tied to acknowledgments in that each round of data 
transfer continues regardless of any particular client' s reccp- iq 
tion problems and/or regardless of any link delay issues 
(e.g., a difference in the time it takes a packet to travel from 
the server to a certain client and the time it takes a packet to 
travel from the server to another different client). Also, each 
subsequent round of tansmission only involves the sending 
of packets which were not received, or received in enor, 
during the previous round, and therefore the server generally 
does not ever need to send the entire file more than once. It 
is reliable because it strives to provide each client with every 
packet, and the reception problems of any individual client 
generally docs not affect the other clients* reception speed ^ 
and accuracy. 

In a preferred embodiment of the invention, the entire 
amount of data to be transferred (e.g., a file) is separated into 
a plurality of blocks, where each block includes a plurality 
of packets. The server completes a round when it finishes 
transmitting all blocks (e.g., the entire file). After a complete 
block has been transmitted, the clients send their acknowl- 
edgments back to the server via a return unicast communi- 
cations path. Block boundaries trigger the sending of 
acknowledgments by the clients. As the acknowledgments 
are coming into the server from the clients for block N, the 
server is transmitting block N+1 (or a subsequent block) out 
to the clients or the server has finished transmitting all of the 
blocks. 

35 

It is noted that the terms *packet\ 'datagram*, and 'frame* 
are used interchangeably herein to identify the same thing, 
namely a unit of data or information which may have a 
source and destination address as part thereof and which is 
sent across the link. 4q 

The foregoing and other objects, aspects, features, and 
advantages of the invention will become more apparent from 
the following description and from the claims. 

BRIEF DESCRIFnON OF THE DRAWINGS 

45 

In the drawings, like reference characters generally refer 
to the same parts throughout the different views. Also, the 
drawings are not necessarily to scale, emphasis instead 
generally being placed upon illustrating the principles of the 
invention. 50 

FIG. 1 is a flowchart of data transmission operations 
according to the invention. 

FIG. 2 is a diagram of a physical configuration which 
allows a server to communicate with one or more clients. 

FIG. 3 is a diagram showing the location of an embodi- 
menl of the invention in relation to the TCP/IP protocol 
stack. 

FIG. 4 is a diagram of a **first pass'* block and frame 
transmission and acknowledgment process according to the gQ 
invention. 

FIG. S is a simplified block diagram of a server in which 
at least a portion of the present invention can be embodied. 

DESCRIPTION 

65 

Referring to FIGS. 1 and 2, in accordance with the 
invention, quick and reliable data transmission fit)m a 



4 

source or server 20 to one or more recipients or receivers or 
clients 22|, 22^, . . . , 22^^ over a communications link 24 
comprises (step 10) transmitting the data (e.g., a file), which 
is in the form of a plurality of frames, over the link 24 to one 
or more of the recipients 22 until the entire file (i.e., all of 
the plurality of frames) have been transmitted over the link 
24. As the frames are being transmitted, frame acknowledg- 
ments from one or more of the recipients 22 are received via 
the link 24 (step 10). If, after the entire file has been 
transmitted over the link 24, the acknowledgments indicate 
that certain frames need to be retransmitted over the link 24 
(step 12), only those certain frames are retransmitted (step 
14). As those certain frames are being retransmitted over the 
link 24, frame acknowledgments from one or more of the 
recipients 22 are received via the link 24 (step 14). This 
process is then repeated as many times as necessary until no 
more frames need to be retransmitted, as indicated by steps 
14 and 16. Alternatively, as indicated in step 16, the process 
can be stopped either when the number of passes through 
step 14 equals a maximum allowable value or when a 
maximum time is met or exceeded. If it is determined at step 
12 or step 16 that no franies need to be resent, the process 
ends (which ending could occur before MAX is reached). 
The initial transfer of die entire file and each of the subse- 
quent transmissions of error frames are generally referred to 
herein, as a "round'* or **pass". 

In tile first pass, Uie server 20 preferably either broadcasts 
the file to all of the clients 22 or molticasts it to a subset of 
all of the clients 22. At least two of the clients 22 typically 
have a different server-to-client frame transmission delay 
associated therewith. Data transmission according to the 
invention is uneffected by such delay differences even if 
significant and even if every client 22 has a different delay 
associated therewith. 

The link 24 can be a computer network (e.g., a LAN, a 
WAN, the Internet), a wireless network (e.g., a cellular data 
network), some combination of these two types of commu- 
nication mediums, or some other conununication medium. 
The plurality of frames transmitted over the link 24 during 
the first round can togetiter represent a computer data file 
being transferred from tiie server 20 to one or more of tiie 
clients 22. 

The server 20 and the clients 22 can be computers, such 
as PCs or workstations, running any one of a variety of 
operating systems including DOS. Referring to FIG, S, tiie 
server 20, regardless of what type of computer it is, typically 
includes a central processor 50, a main memory unit 52 for 
storing programs and/or data, an input/output controller 54, 
a network interface 56, one or more input devices 58 such 
as a keyboard and a mouse, a display device 60, a fixed or 
hard disk drive unit 62, a floppy disk drive unit 64, a tape 
drive unit 66, and a data bus 68 coupling these components 
to allow communication therebetween. Each of the client 
computers 22 generally includes all or some of the compo- 
nents included in the server 20 of FIG. 5. 

In some embodiments, one or more computer programs 
define the operational capabilities of the server 20 and tiie 
clients 22. The programs can be loaded into tiie server 20 
and tiie clients 22 via the hard drive 62, the floppy drive 64, 
and/or the tape drive 66. Alternatively, the programs can 
reside in a permanent memory portion (e.g., a ROM chip) of 
the main memory 52. In some other embodiments, the server 
20 and/or the clients 22 can include specially-designed, 
dedicated, hard-wired electronic circuits, which perform all 
functions described herein without the need for instructions 
from computei* programs. The invention can be used, for 
example, to load quickly and reliably new revision levels of 
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the client software electronically from the server onto one or 
more of the clients. 

Referring to FIG. 3, the invention preferably operates at 
the application layer 30 of the TCP/IP protocol stack 32 on 
top of UDP The invention also could operate at the appli- 5 
cation layer above the connectionless transport layer present 
in other protocol stacks such as IPX in the NetWare SPX/ 
IPX protocol suite. UDP stands for User Datagram Protocol, 
and it is the TCP/IP standard protocol that allows an appli- 
cation program on one computer to send a datagram to an 
application program on another computer. UDP uses the 
Internet Protocol (IP) to deliver datagrams. UDP datagrams 
differ from IP datagrams in that UDP datagrams include a 
protocol port number which allows the sender of the data- 
gram to distinguish among multiple destinations (i.e., appH- 15 
cation programs) on the receiving computer UDP datagrams 
also typically include a checksum for the data being sent. 

In general, data transmission according to the invention 
includes four aspects: IDLE, ANNOUNCE/REGISTRA- 
TION, TRANSFER, and COMPLETION. In the IDLE state, ^ 
there is no activity. When a collection of data (e.g., a file) is 
selected for transmission by the server 20, the 
ANNOUNCE/REGISTRATION phase is entered. During 
any of the four phases, all files are available to an operator 
at the server 20. ^ 

ANNOUNCE/REGISTRATION 

In this phase, the server ANNOUNCES to the clients that 
a file is about to be transferred and provides the parameters 
associated with the transfer of the file. The maximum 
duration of this phase is expressed in minutes, and it is 
configurable. 

Clients are obhged to register with the server that they 35 
received an ANNOUNCE message. When a client sees the 
ANNOUNCE message, the client verifies that it is associ- 
ated with the group identified in the message. It is implicit 
in the receiver being able to process the ANNOUNCE 
message that the receiver has a correct server IP address and 40 
a correct port number. The clients automatically respond to 
ANNOUNCE packets with REGISTRATION packets until 
they see their address in a registered client list in a subse- 
quent ANNOUNCE packet. The REGISTRATION packet 
acts as a positive acknowledgment to the server about the 45 
client's participation. Once the server receives the client's 
REGISTRATION packet, the server adds the client to the 
client list in the next broadcast of the ANNOUNCE packet. 
The client list is maintained by the server, but it is not shown 
in any of the drawings. When the client receives an 50 
ANNOUNCE packet with the client's ID in the client list, 
registration for the client is complete. When all expected 
receivers have responded to the ANNOUNCE message, the 
announce time remaining will be set to zero and actual 
transmission of the file will begin after a final announce 55 
interval. This registration indicates that the client can par- 
ticipate in the group, as it has the resources to handle the file 
about to be sent. To prevent unwanted participation, encryp- 
tion key exchange can take place at group setup. 

Once file transfer begins, ANNOUNCE packets continue 60 
to be sent. This allows for the possibility of late registration. 
Because coordination of start can be hampered by many 
uncontrolled aspects, even differing time zones, participa- 
tion should be allowed as long as the group would not be 
appreciably impeded by the late registration. If a client starts 65 
participation just after the file transfer begins, the impact on 
the group would be minimal and registration should be 
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allowed. ANNOUNCE packets continue to be sent until any 
client responds with a COMPLETION packet COMPLE- 
TION is described hereinafter. If clients are close to comple- 
tion, adding a new client would delay completion of the 
group by more than an acceptable period. For example, if a 
perfect transfer takes two hours, then one pass through the 
file would probably be successful for one client at least. A 
late registration would require anyone else in the group to 
wait potentially for a full transfer. The group members may 
only need to wait for considerably less to complete. Also, 
rejection of late transfer may allow the server to create a list 
of clients that can perform the transfer at a later time with a 
unicast transmission. 

Throughout this period, ANNOUNCE packets will be 
transmitted at an interval which is configurable. 

All the characteristics of the transmission are transmitted 
in the ANNOUNCE packet. 

On receiving this ANNOUNCE message, the client 
responds with a unicast datagram to the server. The response 
indicates whether or not the receiver has the facilities to 
receive the file. It also indicates, in the caise of an aborted 
transmission, whether the client has enough context to 
resume the transmission. The duration of the announce 
period in some instances should allow for an operator at the 
server site to initiate a call to the client site indicating that 
the computer is either not available or does not have the 
facilities for the transfer. At the client site, the corrections 
could be made either manually or, if so configured, under 
remote control from the server to free up resource so it can 
participate in the transfer. 

At any point in time throughout the transmission, the 
cUent may respond to this packet indicating that they aborted 
the transmission from their end indicating the reason in the 
message. 

TRANSFER 

Upon entering the data transfer phase, a transmission log 
is initiated and maintained at the server. This log. however, 
is not shown in any of the drawings. Each of the cUents also 
maintains a transmission log, but it also is not shown in any 
of the drawings. The log maintained at each of the clients is 
mentioned hereinafter under the "COMPLETION** heading. 

As flies having 50 to 100 megabytes of data or more can 
be transferred, holding the entire file in memory at the server 
for the extent of the transfer generally is unrealistic. The 
number of clients which arc to receive the file can be one 
thousand or more, and thus waiting for acknowledgments 
from each of them before continuing on to the next block 
transfer is unacceptable. 

The server logically breaks each file to be transferred into 
blocks of frames, and each block typically includes a plu- 
rality of frames and possibly thousands of frames. Referring 
to FIG. 4, in one example, the server 20 has broken a file into 
four blocks, namely. Block 1, Block 2, Block 3, and. Block 
4, wherein each block includes one or more frames. Each 
block represents a unit that will be positively or negatively 
acknowledged by every client participating in a transfer 
when the client determines that a block has been sent by the 
server. The client detects this by a change in block number 
in data packets received, because each frame sent indicates 
its block number and its firame number vwthin that block. 
Breaking the file into blocks provides at least two advan- 
tages: (i) decreasing the number of acknowledgments 
required and (ii) reducing the memory requirements in the 
server for determining next file pass transfer blocks. 
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Data transfers are not directly lied to acknowledgments. 
Transfer continues regardless of missed acknowledgments 
or previously missed data packets by any individual client. 
This allows simplicity of design and ensures that individual 
client problems provide minimal impact on the group as a 
whole. Note also that clients arc responsible for sending 
block acknowledgments based on what they hear from the 
server. 

Referring to RG. 4, the server starts the transfer by 
sending the first frame of the first block (i.e., the first frame 
of Block J ). The server sends the frames at a rate that is 
configurable. This represents the basic transfer rate that may 
be throttled back (i.e., decreased) based on performance. 
The server continues sending the frames of the file until the 
complete file has been sent once into the network (i.e., until 
Blockj through Block^ are sent). This is defined as the first 
pass or first round, and it takes an amount of. time repre- 
sented in FIG. 4 as "B4", Some clients may receive the 
complete file (i.e., all four blocks) correctly after the first 
pass, in which case they have finished receiving the file. 
Clients receiving one or more frames in enor, or not 
receiving one or more frames at all, require the rescnding of 
certain "pieces" of the file (i.e., the erroneously-received or 
missed frames) on subsequent passes or rounds. Each sub- 
sequent pass requires the transmission of fewer frames 
because only frames not received (or received in enor) in the 
previous round get retransmitted in the next round. 

The maximum pass count or the maximum time to 
complete (RG. 1 ) is a configurable parameter. There may 
be clients that have not received all of the file correctly by 
the time of the maximum pass or the maximum time 
duration. These clients are identified by the server, and the 
server can take further action to get these clients the rest of 
the information via, for example, a unicast file transfer 
process. 

As the server passes block boundaries (i.e., Bj, Bj, B3 and 
B4 in FIG. 4), the individual clients send **block positive" 
acknowledgments ("Ack" in FIG. 4) and "selective reject 
negative" acknowledgments CNak" in FIG, 4) for each 
block. The acknowledgments from the clients for each block 
are received at the server sometime after the boundary of 
that block is passed. A "block positive" acknowledgment for 
a particular block means that particular block was received 
correctly in its entirety at the client. A "selective reject 
negative" acknowledgment for a particular block means that 
some or all of the frames in that particular block were 
received in error, or were not received at all indicating that 
the network did not deliver them. Because of the possibility 
of frames being received out of order, a dropped frame is 
declared if five or more frames after it are received and it is 
not. Thus, selective reject negative acknowledgments (also 
referred to herein as negative acknowledgments) indicate 
which frames were received in error or not received. 

On subsequent passes (i.e., after the first pass shown in 
FIG, 4), clients only respond with acknowledgments for 
blocks not previously received correctly on previous passes. 
Since the server sends pieces (frames) of the file needed by 
various clients to all clients in subsequent passes, many of 
the clients will have already received it corrccdy on the first 
pass and do not have to acknowledge reception again. 

In general, all information returning back to the server 
from die clients may be transmitted on a return path which 
is separate from the path(s) which the server uses to transfer 
the frames to the clients. However, for the purposes of this 
description, the communications link 24 (FIG. 2), or other 
path which allows the server and the clients to communicate. 
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should generally be taken to mean both the server-to-client 
link and the return client-to-server link. 

The server maintains various information about the trans- 
fer and the participants in the transfer. In the preferred 
embodiment, this information is maintained by the server in 
the form of data structures or lists. The server maintains and 
uses this information to record and determine the status of 
the file transfer. 

Referring to Table 1, a client status structure is maintained 
by the server. This client stams structure includes a list of the 
status of the participants of the broadcast or multicast group 
based on data from the announce registrations that are 
received by the server. (Announce registration is described 
herein, for example, under the ANNOUNCE/REGISTRA- 
TION heading). The chent status list can indicate, for 
example, unregistered clients and aborted clients. The client 
status list generally includes an entry for each client 
included in the broadcast or multicast group, and die number 
of clients which can be tracked in die client status list can 
number in the tiiousands. For example, it might be, as shown 
in Table 1, that the first client (e.g., client 22i of FIG. 2) has 
aborted while most other clients in the group are registered. 

TABLE 1 

Client Status 

Clicni 1 - Aborted 

Qient 2 - Registered 

Cliem 3 - Registered 

Qient 8 - Regtstersd 
Clieot 12 - Registered 
Client 13 - Registered 



CHent N - Registered 



TABLE 2 



Frame Status 



40 



45 



50 



55 



60 



Block 1, 
Block 1, 
Block 2. 
Block 2, 
Block 2, 
Block 9, 
Block 9, 



frame 20 
ftame 25 
Frame 10 
frame 22 
Frame 24 
Frame 3 
Frame 5 



TABLE 3 



Lost Heard 



Client I • XA seconds 
Cliem 2 - XB seconds 
Client 3 - XC seconds 
Cliem 8 • XD seconds 
Client 12 • X£ seconds 
Client 13 -XF seconds 



Cliem N • ZZ seconds 



Note that the subscript "N** is used here and in other 
places in this description, but it docs not necessarily repre- 
65 sent the same integer each time it i s used. For example, there 
are not necessarily the same number of clients shown in HG. 
2 as there clients listed in the tables above. 
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Referring to Table 2, the server also maintains a frame 
data stiucture which indicates all selective rejects on indi- 
vidual frames from all clients. If multiple clients missed the 
same frame, the frame data structure would indicate only 
that the frame was missed. That is, the frame data structure 5 
is not maintained on a client-by-client basis by the server. It 
generally is undesirable for the server to maintain a list of 
missed frames on a per client basis because such a scheme 
would use an inordinate amount of memory, particularly 
when a large number (e.g.. one thousand or more) of clients lo 
are involved in the broadcast or multicast. For example, it 
might be. as shown in Table 2, that one or more of the clients 
either did not receive or received in error frames twenty and 
twenty-five of Blocki, the three frames indicated of Block^, 
fi^e one of Block*,, etc. If the frame status maintained by 15 
the server indicates that a particular frame of a particular 
block needs to be retransmitted, it will be true that at least 
one of the clients has not acknowledged successful comple- 
tion of that particular block. After the server has sent the 
entire file once, the server would then pass through the frame 20 
status information and resend only the frames listed therein. 
This would continue, pass after pass, until all clients had 
positively acknowledged all blocks and the frame status list 
is empty (or the maximum number of rounds, or maximum 
time, had been reached as indicated in FIG. 1). 25 

A third piece of information that the server maintains 
about the file transfer, as shown in Table 3, is a LAST__ 
HEARD_FROM time indication for each client. For 
example, the server might not have heard from the first client 
(e.g., client 22, of FIG. 2) for XA seconds, the second client 30 
(e.g., client 22^ of FIG. 2) for XB seconds, and the last client 
(e.g., client 22^) for ZZ seconds. Also, each client maintains 
a similar LAST_HEARD_FROM time indication for the 
server, but this client list is not shown in any drawing or 
table. Maintenance of this parameter allows either end to 35 
abort participation if the other end has dropped out for a 
period of time which exceeds a configurable timeout period. 

Note that for any given pass, if any acknowledgment docs 
not get back to the server, the client will send back to the 
server the same reject and retransmission request messages 
during the next pass by the server. This means that if a 
certain client is not being heard by the server, that client will 
have to participate longer but that client will not appreciable 
impact the rest of the receiving clients. 

A fourth piece of information stored at the server is 
statistics on the broadcast or multicast group. When a 
transmission is completed, summary information is provided 
on the transmission that can aid an operator in determining 
system performance problems and/or the performance prob- -q 
lems of a particular client. 

In addition, the server gels a good indication of conges- 
tion by the number of acknowledgments not getting back, 
and the server can then throtUe back (i.e., decrease) the file 
transfer rate. When the acknowledgments arrive at a better 55 
rate, the server can throttle forward again. The invention 
thus provides simple and eflfective congestion control. 



10 



Asynchronous Selective Acknowledgments 



60 



Asynchronously throughout the transmission, the server 
performs a process which selectively queries the clients for 
block acknowledgments which have not been received by 
the server with unicast packets. The frequency of these 
requests is a configurable parameter. ^ 

This asynchronous process from the server will send 
unicast selective ack request messages to solicit responses 



from individual clients which have not responded. The 
responses are in the form of a standard response message. 
They will contain the range of blocks to acknowledge up to 
the block requested along with a bitmap describing the 
specific block in error if there are errors to report. 

Multiple Passes Through tiie File 

Once the file has been completely processed once (i.e., 
after the first pass or round), the transmission process 
according to the invention will increment a pass couiiter and 
then scan the frame status list (Table 2) for die first block in 
which there was an error. Upon finding this first-error block, 
die server will resend Uic missed packets in that block. 
Acknowledgments for these missed packets will, as 
described previously, be generated by the clients when they 
detect an error in a block. This is consistent with the first 
pass. All block positive acknowledgments and selective 
reject negative acknowledgments are indications of state and 
are therefore not specific to a pass though they may change 
with each pass. In a preferred embodiment, the selective 
reject negative acknowledgments are in the form of bitmaps 
where Uie entire word represents a block and each bit in that 
word represents a different one of the frames which make up 
that blocL 

Transmission Abort 

If, during the transmission, a fault is encountered which 
cannot be rectified, or if the operator manually aborts, a 
transmission abort sequence will be initiated. This sequence 
entails the repeated transmission of an Abort message for a 
certain interval (e.g., for an interval which is specified in a 
transmissions file). The receivers acknowledge the abort 
message and can take action to. for example, either save the 
context for a potential resumption of the transfer or reini- 
tialize the context to prepare for another transmission. 

There is a facility which allows the user to initiate a 
transmission abort A reason code can be set to cither 
suspend or initialize. In former case, the transmission may 
be resumed at a later time, and, in the latter case, the clients 
will be requested to reinitialize their contexts. 

COMPLETION 

The server detects completion of individual clients by 
receiving a positive block acknowledgment for all blocks. 
The client knows it's done as soon as it has all blocks of the 
file, but the client must continue to send acknowledgments 
until the server confirms completion. 

When the server determines that a client has completed 
reception of die file, the server sends an ANNOUNCE^ 
COMPLETION packet. This packet replaces die original 
ANNOUNCE packet and also signals the end of late regis- 
tration. Once a client receives the announce completion 
packet, tiiat client can stop participating in block acknowl- 
edgments. The client will then update its transmission log to 
indicate that the transfer has been successftiUy completed. 

An ability to abort a transfer from die server or client is 
included. An abort packet provides the server and client with 
die ability to prematurely abort a transfer. If the client sends 
an abort, die server initializes die client until the client 
reregisters. If the server aborts, the transfer can be restarted 
widiout sending die full file on the first pass by having die 
server make one pass requesting block status from all clients 
and dien starting with the returned rejections. The clients 
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read the previous transfer and determine which blocks were 
missing or in error. 

When a single pass has been made through the file, certain 
chents will be considered to have completed. As each 
completes, they will generate a complete message to the ^ 
server. The server will regularly send out a transmission 
complete message which references the addresses that the 
server recognizes are complete. 

10 

Congesdon 

If the packet error rate for die block exceeds a threshold 
level of errors per block in aggregate across the set of chents 
or if there are specific packets which are dropped by a 
configurable percentage of the clients, then congestion is 
indicated These two factors comprise an error level in the 
transmission. Initially, however, the error level will be 
represented by the gross percentage of frame errors reported 
per block. In managing congesdon. the transfer rate will be ^ 
stepped down until the error level as indicated by negative 
acknowledgments shows improvement. 

To avoid bouncing the transfer rate after each block, the 
congestion algorithm should allow for 2 congestion error 
levels. The first is the threshold for invoking congestion, and 23 
the second represents the error level to which the transmis- 
sion should improve to before turning congestion manage- 
ment ofiF. 

The server will manage congestion by throttling back on 
the rate at which it sends out frames. It will step down the 30 
transfer rate each time it detects congestion. Hie step down 
will take the congestion level which ranges from 1 to 8, 
multiply it by 2, and then divide the current transfer rate by 
the product (i.e.. "Current Transfer Rate" divided by "Con- 
gestion Level * 2"). Note that a congestion level of zero 35 
indicates no congestion. 

Multicast 

Multicast can be in two forms: pseudo-multicast where ^ 
die network still delivers data to the entire broadcast group, 
and multicast IP where the network routes traffic based on 
multicast routers and Internet specification RFC 1112 is 
implemented in the clients. 

In both cases, multicast groups are set up under initiation 
of the server. The server sends notifications on a unicasi 
basis to clients to inform them of membership in a particular 
multicast group. These multicast groups can be set up and 
dismanUed rapidly, allowing for a dynamic configuration of 
multicast groups. For example, a multicast group could be 
set up to be only in place for the transmission of a particular 
file, after which time the group was dismantled. 

With pseudo-multicast, the network still delivers traffic on 
a broadcast basis, but clients not in the group discard the data 55 
not destined for it. When the group is set up, security keys 
are also disseminated so that clients outside the group cannot 
read the data even if it happened that the data was not 
discarded at that node. Also, with pseudo-multicast, the IP 
address remains 255.255.255.255 which is the normal IP 
broadcast address. As with broadcast, this address becomes 
mapped to a broadcast DLCL A multicast header is selected 
for the group and becomes the group differentiator. 

With multicast IP, the network is a router network where 
the routers suppon Class D multicast IP addresses and 65 
multicast routing. The chents support RFC 1112. "Host 
Extensions for IP Multicasting**. RFC 1 1 12 provides for host 
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notification of their presence to the nearest multicast router 
for the purpose of update of router tables. 

A functional description of the above-described invention 
is provided below. 

Referring back to FIG. 2, a purpose of the invention is to 
enable the simultaneous transmission of small or large data 
files (e.g., files up to 100 megabytes or more in size) by a 
server 20 to up to 1000 or more receiving nodes 22 over a 
wide area network (WAN) connection 24. The invention also 
is able to work over local area networks and other types of 
communications links, as described previously. The trans- 
mission medium 24 can be a broadcast frame relay network 
provided by a carrier. The receiving nodes 22 on the network 
24 can be located on an Ethernet LAN and attached to a 
router. The broadcast group can be addressed using the 
255.255.255.255 broadcast IP address which can be mapped 
to die broadcast Ethernet address, which can be mapped to 
a broadcast frame relay DLCI. The router in diis case can act 
as a bridge for tins traffic. Alternatively, the server can be 
interface directly to a broadcast frame relay DLCI, 

Multicast can be supported in two ways: pseudo-multicast 
and multicast IP, as mentioned previously. 

Files to be transferred to the clients can be loaded onto die 
server 20 via tape (e.g., the tape drive 66 of FIG. 5) or, if die 
files are small enough, by floppy (e.g.. die floppy drive of 
FIG. 5). Also, files to be transferred can be loaded onto the 
server 20 via FTP (File Transfer Protocol) from the source 
of the file over a LAN or other network, for example. Hie 
files generally can be in any format. The data file is then read 
in from the tape or floppy into a file system of the trans- 
mission server 20. Note that the server 20 must have 
sufficient space available to read in an uncompressed copy 
of the data file. For pseudo-multicast service, the data file 
also can be encrypted so that noneligible receiven cannot 
receive and use die data file. Each transmission file prefer- 
ably is imiquely identified. There preferably is an indication 
as to its content and time of generation. The input files to the 
process can be over 100 megabytes in size, and the system 
can also handle files much larger than 100 megabytes. 

The file can then be stored on the server 20 and prepared 
for transmission. Data from previous transmissions will 
need to be readily available on the server 20 for some period 
of time in case they need to be retransmitted. A mechanism 
for accessing the data is provided such that the data can be 
readily queued-up for retransraissioa 

When the file is encrypted, it can be transmitted. For 
efficiency, the file is Uransmitted in blocks. The size of a 
block is derived from the largest packet which can be 
transferred over the conununications path 24. Its derivation 
is based on the fact that the clients will need to indicate to 
the server which of the packets in a block they failed to 
receive. One way, and generally the simplest way, to do this 
is to send a bitmap indicating by a bit setting positionally 
which packets were not received. The size of the block 
therefore is approximately the number of packets which can 
be acknowledged in a bitmap which itself can be contained 
in a packet For example, if the packet size were 256 bytes, 
then the most bits a packet could contain would be 
256(bytes/packet) * 8 (bits/Byte)=2048 (bits/packet) which 
means that the largest allowable block size would be a block 
having 2048 packets. 

Although receiving nodes 22 can be interfaced to an 
Ediemet LAN at 10 Mbps, the WAN ft^e relay links are 
often of much lower speeds than that. Thus, an explicit 
transmit data rate is seltable/configurable. and an automatic 
flow control is provided in the event of congestion, as 
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mentioned previously. The user is alerted when congestion 
is invoked. 

Receiving nodes can each experience resource problems 
either prior to or during a transmission. Receiving nodes are 
enabled to query their resources prior to a transmission and 5 
determine if they have the facilities to receive the data. If 
not, then they should cither reinitialize space which is 
dedicated for the transmission or should indicate that they 
cannot participate in the transmission and corrective mea- 
sure can be undertaken through different channels. A facility 
is provided where the server can force the availability of disk 
space remotely, to allow the U-ansfer of the file to take place. 

The receivers 22 must also be aware of what they are 
listening for. When a datagram is received on a dedicated 
channel, the node 22 must determine if it is being addressed, 
An issue can arise when this application is being used by 
more than one transmission server 20. There must be a way 
of guaranteeing that a receiving node 22 is participating in 
exactly one transmission at a given time. By dedicating a 
UDP port to a server 20 and also relating an encryption key 
to that server, it is ensured that a receiving node employing ^ 
a promiscuous mode tap on the network 24 will not have the 
ability to be able to interpret the transmitted data. 

Some reference information is maintained on the trans- 
mission server 20. There preferably is a list of all the 
potential receiving nodes in the network. Enough reference ^ 
information preferably is available to allow the information 
provider to manage the clients in the case of service failures, 
problems, etc. There preferably will be a transmission data- 
base where an encrypted compressed data file is maintained 
ready for transmission. The transmission database contains 30 
the prepared data along with descriptive information of up 
to, for example, 70 bytes identifying the content of the files. 

Each transmission preferably has a completion status 
indicator record and a log of all errors encountered during 
the transmission. There preferably also is an event file with 35 
a hst of all the nodes for which the transmission failed, who 
to call, and why it failed. 

At any point in time during the transmission, an operator 
is able to interrogate the status of the transmission as it 
applies to the server 20 and each of the receiving nodes 22. 40 
Alerts are generated if there are problems communicating to 
certain clients or other problems. If any intervention is 
indicated, the operator is allowed to initiate the corrective 
action. 

For ongoing maintenance and management of the service, 45 
the operator is enabled to maintain the list of receivers, 
transmission groups, transmission file descriptors, transmis- 
sion parameters, and transmissions database. A background 
process will maintain the environment and both age data and 
delete it according to housekeeping parameters, if enabled 5Q 
by an alerted operator. 

Variations, modifications, and other implementations of 
what is described herein will occur to those of ordinary skill 
in the art without departing from the spirit and the scope of 
the invention as claimed. Accordingly, the invention is to be 
defined not by the preceding illustrative description but 
instead by the following claims. 

What is claimed is: 

1. A method for transmitting data over a conmiunications 
link, comprising: 

(A) partitioning the data into a plurality of blocks, each 
block including a plurality of frames; 

(B) transmitting all of the frames to one or more recipi- 
ents; 

(C) during transmission, receiving acknowledgments 65 
from the recipients which include indications of frames 
requiring retransmission; and 
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(D) repeating steps (B), (C), and (D) for only those frames 
which the acknowledgments indicate require retrans- 
mission. 

2. A method for quickly and reliably transmitting data to 
at least two recipients over a communications link compris- 
ing: 

(A) transmitting a plurality of firames of data over the link 
to the recipients until all of the plurality of frames have 
been u-ansraittcd; 

(B) while performing step (A), receiving acknowledg- 
ments from one or more of the recipients, the acknowl- 
edgments including indications of frames requiring 
retransmission; and 

(C) after all of the plurality of frames have been trans- 
mitted, repeating for a predetermined number of times 
steps (A), (B), and (Q for only those frames which the 
acknowledgments indicate require retransmission. 

3. A method for quickly and reliably transmitting data to 
at least two recipients over a connmunications link compris- 
ing; 

(A) transmitting a plurality of frames of data over the link 
to the recipients until all of the plurality of frames have 
been transmitted; 

(B) while performing step (A), receiving acknowledg- 
ments from one or more of the recipients, the acknowlr 
edgments including indications of frames requiriiig 
retransmission; and 

(C) after all of the plurality of frames have been trans- 
mitted, repeating until a predetermined amount of time 
has passed steps (A), (B), and (C) for only those frames 
which the acknowledgments indicate require retrans- 
mission. 

4. A method for quickly and reliably transmitting data to 
at least two recipients over the Internet, comprising: 

(A) transmitting a plurality of frames of data over the 
Internet to the recipients until all of the plurality of 
frames have been transmitted; 

(B) while performing step (A), receiving acknowledg- 
ments from one or more of the recipients, the acknowl- 
edgments including indications of frames requiring 
retransmission; and 

(C) after all of the plurality of frames have been trans- 
mitted, repeating steps (A), (B), and (C) for only those 
frames which the acknowledgments indicate require 
retransmission. 

5. A method for quickly and reliably transmitting data to 
at least two recipients over a cellular network, comprising: 

(A) transmitting a plurality of frames of data over the 
cellular network to the recipients until all of the plu- 
rality of frames have been transmitted; 

(B) while performing step (A), receiving acknowledg- 
ments from one or more of the recipients, the acknowl- 
edgments including indications of frames requiring 
retransmission; and 

(C) after all of the plurality of frames have been trans- 
mitted, repealing steps (A), (B), and (C) for only those 
frames which the acknowledgments indicate require 
retransmission. 

6. A method for quickly and reliably transmitting data to 
at least two recipients over a communications liric, com- 
prising: 

(A) grouping a plurality of frames of data into a plurality 
of blocks, eadi block including a plurality of the frames 
and the number of blocks being less than the number of 
frames; 
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(B) transmitting each of the frames of each of the blocks 
over the link to the recipients until all of the plurality 
of blocks have been transmitted; 

(C) while performing step (B), receiving acknowledg- 
ments from one or more of the recipients, the acknowl- 5 
edgments including indications of frames requiring 
retransmission; and 

(D) after all of the plurality of blocks have been trans- 
mitted, repeating steps (B), (C), and (D) for only those 
frames which the acknowledgments indicate require 
retransmission. 

7. The method of claim 6 wherein step (C) comprises 
receiving the acknowledgments wherein each acknowledg- 
ment comprises either a positive acknowledgment or a 
negative acknowledgment^ the positive acknowledgment 
indicating that a particular one of the blocks was received 
correcdy in its entirety by a particular one of the recipients, 



the negative acknowledgment indicating that a particular 
one of the recipients requires that one or more of the frames 
from a particular one of the blocks be retransmitted. 

8. The method of claim 7 wherein step (C) further 
comprises receiving the acknowledgments for a particular 
one of the blocks only after all of the frames of that 
particular block have been transmitted over the link. 

9. The method of claim 8 wherein step (B) further 
comprises transmitting the frames at a predetermined rate. 

10. TTie method of claim 9 further comprising determining 
which, if any, recipients are not acknowledging transmis- 
sions and then adjusting the predetermined rate based on that 
determination in order to increase recipient transmission 
acknowledgment. 
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ABSTRACT 



A data transmission method quickly and reliably transmits 
data (e.g., a file) from a source to one or more recipients over 
a communications link. The method includes transmitting 
the data, which are in the form of a plurality of frames, from 
the source over the link to one or more of the recipients until 
all of the plurality of frames have been transmitted over the 
link. While the data are being transmitted by the source, 
acknowledgments from one or more of the recipients are 
received by the source. The acknowledgments indicate 
which frames require retransmission. After all of the frames 
have been transmiued over the link, a retransmission is 
performed by the source for only those frames which the 
acknowledgements indicate as requiring retransmission. 
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REEXAMINATION CERTIFICATE 
ISSUED UNDER 35 U.S.C. 307 

THE PATENT IS HEREBY AMENDED AS , 
INDICATED BELOW. ^ 

Matter enclosed in heavy brackets [ ] appeared in the 
patent, but has been deleted and is no longer a part of the 
patent; matter printed in italics indicates additions made 
to the patent. 

AS A RESULT OF REEXAMINATION, IT HAS BEEN 
DETERMINED THAT: 

Qaims 6-8 are cancelled. 

Qaims 1-5, 9 and 10 are determined to be patentable as 
amended. 

1. A method for transmitting data over a communications '^^ 
link, comprising: 

(A) partitioning the data into a plurality of blocks, each 
block [including] comprising a plurality of [frames] 
packets and each packet comprising one or more bits, 
the partitioning comprising setting the number of pack- 
ets per block to be approximately equal to the maximum 
number of available bits per packet\ 

(B) transmitting, block by block, all of the [frames] 
packets in each of the blocks to one or more recipients; 

(C) during transmission, receiving, [acknowledgements 
from the recipients which include indications of frames 
requiring retransmission], from one or more of the 
recipients, indications of packets in the transmitted 
block that require retransmission, each indication asso- 35 
dated with a different one of the recipients and com- 
prising a bitmap contained within one of the packets, 
wherein the bitmap comprises a number of bits equal to 
the maximum number of available bits in the packet, 
and each bit of the bitmap represents a different one of 4q 
the packets in the transmitted block; and 

(D) [repeating steps (B), (C), and (D) for] retransmitting 
only those [frames which the acknowledgements indi- 
cate require] packets requiring retransmission by 
repeating steps (B), (C), and (D). <s 

2. [A method for quickly and reliably transmitting data to 
at least two recipients over a communications link compris- 
ing: 

(A) transmitting a plurality of frames of data over the link 
to the recipients until all of the plurality of frames have 
been transmitted; 

(B) while performing step (A), receiving acknowledg- 
ments from one or more of the recipients; the acknowl- 
edgments including indications of frames requiring 
retransmission; and 

(C) after all of the plurality of frames have been 
transmitted, repeating for] The method of claim I 
wherein step {D) comprises repeating steps (B), (C), 
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and (D) a predetermined number of times [steps (A), 
(B), and (C) for only those frames which the acknowl- 
edgments indicate require transmission]. 

3. [A method for quickly and reliably transmitting data to 
at least two recipients over a communications link compris- 
ing; 

(A) transmitting a plurality of frames of data over the link 
to the recipients until all of the plurality of frames have 
been transmitted; 

(B) while performing step (A), receiving acknowledg- 
ments from one or more of the recipients, the acknowl- 
edgments including indications of frames requiring 
retransmission; and 

(C) after all of the plurality of frames have been trans- 
mitted repeating until] The method of claim 1 wherein 
step (D) comprises repeating steps (B), (€% and (D) for 
a predetermined amount of time [has passed steps (A), 
(B), and (C) for only those frames which the acknowl- 
edgments indicate require transmission]. 

4. [A method for quickly and reliably transmitting data to 
at least two recipients over] The method of claim 1 wherein 
the link includes the Interne t[, comprising: 

(A) transmitting a plm-ality of frames of data over the 
Internet to the recipients until all of the plurality of 
frames have been transmitted: 

(B) while performing step (A), receiving acknowledg- 
ments from one or more of the recipients, the acknowl- 
edgments including indications of frames requiring 
retransmission; and 

(C) after all of the plurality of frames have been 
transmitted, repeating steps (A), (B), and (C) for only 
those frames which the acknowledgments indicate 
require retransmission], 

5. [A method for quickly and reliably transmitting data to 
at least two recipients over ] The method of claim 1 wherein 
the link includes a cellular network[, comprising: 

(A) transmitting a plurality of frames of data over the 
cellular network to the recipients until all of the plu- 
rality of frames have been transmitted; 

(B) while performing step (A), receiving acknowledg- 
ments from one or more of the recipients, the acknowl- 
edgments including indications of frames requiring 
transmission; and 

(C) after all of the plurality of frames have been 
transmitted, repeating steps (A), (B), and (C) for only 
those frames which the acknowledgments indicate 
require retransmission]. 

9. The method of claim [8] 2 wherein step (B) further 
comprises transmitting the [frames] packets at a predeter- 
mined rate. 

10. The method of claim 9 further comprising determining 
which, if any, recipients are not acknowledging transmis- 
sions and then adjusting the predetermined rate based on that 
determination in order to increase the recipients^ [transmLs- 
sion acknowledgment] receipt of packets. 

* ♦ * ♦ ♦ 
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